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Abstract 

Teachers  need  different  tools  for  constructing 
virtual  realities  than  do  professional  program- 
mers. Teachers  building  tutoring  environments 
need  only  and  should  only  provide  declarative 
and  nonquantitative  specification  of  the  applica- 
tion, as  such  information  is  sufficient  to  build 
powerful  prototypes  or  even  products  when 
exploited  properly.  We  describe  our  METU- 
TOR  tutor-generation  system  for  sequential- 
action  skills,  which  uses  means-ends  analysis 
on  a  teachers' s  declarative  specification  of  a  set 
of  actions.  METUTOR  asks  the  teacher  to 
specify  conditions  for  recommending  actions, 
preconditions  of  actions,  and  expected  and  ran- 
dom consequences  of  actions.  METUTOR  also 
asks  the  teacher  to  associate  pictorial  and/or 
aural  representations  with  facts,  and  to  specify 
how  and  when  to  use  them.  METUTOR  pro- 
vides facilities  for  automatic  resolution  of 
interactions  and  conflicts  between  media 
objects.  We  show  examples  from  a  firefighting 
tutor  and  a  pilot's  emergency  tutor. 

1.  Introduction 

Virtual  reality  can  enhance  a  wide  range  of  educa- 
tional simulations.  For  instance,  military  training 
includes  many  specialized  procedural  skills  for 
emergencies,  e.g.  firefighting.  Broad  educational 
application  of  virtual-reality  ideas  has  been  limited 
by  the  necessary  complexity  of  their  implementa- 
tion. Our  METUTOR  project  tries  to  address  this 
problem  by  providing  special  tools  for  computer- 
inexperienced  teachers  and  instructors  to  develop 
their  own  virtual-reality  tutors  for  procedural  skills. 


An  example  of  our  target  audience  is  an  instructor 
who  trains  Navy  firefighting-team  leaders. 

Computer-inexperienced  teachers  often  have  trouble 
giving  algorithms  for  procedural  skills,  although 
they  can  specify  correct  sequences  of  actions  in  par- 
ticular circumstances.  Furthermore,  they  usually 
cannot  describe  numerically  or  mathematically  the 
components  of  the  virtual  reality,  as  this  often 
requires  sophisticated  mathematics.  This  rules  out 
all  but  minimal  three-dimensional  modeling,  and 
usually  requires  a  discrete  representation  of  time. 
But  if  teachers  can  indeed  perform  the  skill  they 
wish  to  teach,  they  must  know  the  best  thing  to  do 
next  in  any  situation.  This  "local"  knowledge  is 
close  to  the  concept  of  declarative  specification  in 
languages  like  Prolog,  in  which  the  programmer 
specifies  what  things  must  be  accomplished  but  not 
entirely  their  order  [4]. 

So  our  METUTOR  project  has  devised  an  approach 
that  combines  a  simple  language  for  specifying  state 
descriptions  and  discrete  state  changes,  a  simple 
media-object  (bitmap  and  audio)  construction  facil- 
ity, and  a  way  to  relate  state  descriptions  to  media 
objects.  Our  specification  syntax  is  that  of  Prolog, 
and  our  implementation  is  in  Quintus  Prolog.  Our 
state-change  semantics  [4]  is  based  on  means-ends 
analysis  with  several  of  our  own  additions,  and  our 
handling  of  media  objects  is  suggested  by  tech- 
niques of  cartoon  animation  with  several  additional 
features. 

Means-ends  analysis  is  a  problem-solving  technique 
useful  for  a  broad  range  of  human  problems,  and 
people  often  seem  to  use  something  like  it  in  ever- 
day  activity.  To  automate  it,  we  must  identify  a  set 
of  possible  discrete  actions,  and  preconditions, 
postconditions,  and  recommendation  circumstances 
for  each  action.    Mean-ends  specification  of  actions 


permits  a  tutor  to  comment  on  the  appropriateness 
of  student  actions  in  solving  a  problem,  since  it 
knows  the  possible  reasons  for  them.  We  extend 
basic  means-ends  analysis  with  context  dependency 
(so  the  same  action  has  different  preconditions  or 
results  depending  on  other  facts  that  are  true  when  it 
is  applied),  and  most  importanUy,  constrained  ran- 
dom changes  to  the  states  to  provide  an  element  of 
unanticipated  challenge  to  the  student.  All  this 
teacher-supplied  action  specification  is  local  infor- 
mation, and  for  most  applications  will  be  intuitive; 
see  section  3  for  examples.  What  teacher  errors  in 
specification  do  occur,  as  we  have  observed  in  forty 
tutors  written  using  the  METUTOR  system,  are  usu- 
ally errors  of  omission  not  commission,  for  which 
we  have  special  debugging  tools. 

Means-ends  analysis  works  by  repeatedly  comparing 
the  current  state  and  the  goal  objectives,  and  then 
selecting  the  highest-priority  action  recommended 
for  at  least  some  of  the  differences.  If  the  highest- 
priority  action  cannot  be  done  immediately,  a  recur- 
sive subproblem  ("precondition  recursion")  is 
created  with  goal  the  preconditions  of  the  desired 
action.  If  further  differences  between  current  state 
and  goal  objectives  remain  after  the  desired  action 
has  been  performed,  a  new  recursive  subproblem 
("postcondition  recursion")  is  created  for  the  current 
state.  Although  more  sophisticated  methods  of  plan- 
ning have  been  developed  in  artificial  intelligence, 
means-ends  is  surprisingly  broad  in  applicability  and 
robust.  For  instance,  random  occurrences  can  be 
taken  in  stride  because  means-ends  constantly 
recomputes  what  to  do.  Means-ends  can  exploit 
well  the  automatic  backtracking  feature  of  software 
like  Prolog,  so  even  if  the  teacher's  priorities  for 
actions  are  poor,  analysis  will  eventually  back  up 
and  try  something  else.  So  means-ends  is  a  good 
choice  for  tutoring  software  that  computer- 
inexperienced  teachers  must  write. 

Using  means-ends  analysis  means  that  virtual  reali- 
ties constructed  with  METUTOR  are  not  just  pas- 
sive, but  can  advise  the  student.  While  the  student 
thinks,  the  tutor  reasons  hypothetically  to  find  the 
best  sequence  of  actions  to  solve  the  problem  from 
the  current  state.  Then  if  the  student  picks  a  subop- 
timal  action,  the  tutor  can  use  22  domain- 
independent  mal-rules  (of  which  more  than  one 
could  apply  to  the  same  situation),  exploiting  this 
hypothetical  reasoning  plus  automatic  backtracking, 
to  understand  what  the  student  is  doing.  Appropri- 
ate tutoring  is  tied  to  each  mal-rule.  For  instance: 
-If  the  student's  action  does  not  change  any- 


thing, say  so. 

-If  the  original  goal  is  unsolvable,  say  so  and 
stop. 

—If  the  student  has  five  times  avoided  a  cer- 
tain best  action,  give  a  hint. 
—If  the  student's  action's  name  is  similar  to 
the  best  action,  give  a  hint. 
—If  the  student's  action  is  the  exact  opposite 
of  the  best  action,  give  a  hint. 
—If  the  student's  action  only  makes  sense  by 
misreading  the  current  state  slightly,  point  this 
out. 

-If  the   student's   action    is   unnecessary   to 
solve  the  problem,  point  this  out. 
-If  the  student  has  returned  from  a  digression, 
point  out  where  and  how  they  digressed.    (A 
digression  is  signalled  by  an  action  that,  while 
perhaps  eventually  needed,  does  not  make  the 
problem  easier  to  solve.) 
-If  the  student  picks  a  recommended  action, 
but  that  action  is  not  of  the  highest  priority, 
analyze  the  justifications  for  both  actions  in 
terms  of  differences  in  precondition  chains  in 
the  simplest  terms. 
Hypothetical    reasoning   and   mal-rules    have   been 
used  in  other  procedure  tutors,  e.g.  [2],  but  means- 
ends  permits  especially  general  mal-rules.    (Not  to 
be  confused  with  the  student  mal-rules,  30  addi- 
tional rules  check  for  teacher  errors  like  misspelled 
words  and  impossible  preconditions.) 

2.  Specification  of  graphics 

A  central  objective  of  METUTOR  is  to  help  the 
teacher  set  up  a  many-to-many  mapping  between 
facts  describing  the  virtual  reality  and  media  objects 
depicting  it.  That  is,  a  fact  or  facts  will  correspond 
to  media  objects  or  objects,  as  in  the  more  general 
Prolog-based  approach  of  [3].  Teachers  must  obtain 
or  construct  images  and  sounds,  and  we  provide 
tools  to  facilitate  this.  For  audio,  the  teacher  will 
specify  a  set  of  frequencies  and  amplitudes  of  those 
frequencies  as  a  function  of  time  within  a  time 
period,  which  permits  a  variety  of  simple  noises. 
For  images  we  already  provide  a  simple  facility  for 
freehand  line-drawing  using  the  mouse,  plus  the 
capability  of  reusing  bitmap  pictures  from  a  library 
[5].  Regions  can  be  drawn,  moved,  scaled,  clipped, 
filled,  among  other  standard  graphics  operations. 
When  finished,  the  screen  location  of  the  bitmap 
and  its  associated  fact  or  facts  are  associated.  Real- 
istic images  are  not  needed  in  many  tutors  for  adult 
students,  since  such  students  can  understand  meta- 
phoric  symbols;  otherwise,  a  professional  program- 
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mer  could  later  improve  an  teacher's  images  and 
sounds.  Figures  1  and  2  show  example  metaphori- 
cal displays. 

The  mapping  between  internal  facts  and  media 
objects  can  be  many-to-many  when  the  semantics  of 
the  two  is  sufficiently  different.  For  instance,  a  big 
fire  in  a  compartment  on  a  Navy  ship  can  be  shown 
by  bitmaps  of  flames  scattered  across  the  picture. 
Conversely,  multiple  facts  may  correspond  to  a  sin- 
gle image  if  some  facts  are  necessary  to  explain  oth- 
ers. For  instance,  the  facts  that  the  oxygen  has  been 
tested  and  the  oxygen  is  safe  can  be  displayed  in  a 
single  image,  a  dial  reading  "OK". 

An  object-oriented  system,  with  inheritance  and 
defaults,  is  the  obvious  way  to  assign  properties  to 
media  objects.  For  instance  for  a  firefighting  tutor, 
the  instances  of  flames  displayed  on  the  screen  can 
inherit  shape  and  color  from  an  object  representing 
a  single  prototypical  flame,  but  could  override  its 
default  size  and  screen-position  attributes  when  the 
fire  is  almost  out.  In  general,  a  media  object  can 
have  these  inheritable  properties  (some  of  which  are 
not  yet  implemented): 

(1)  a   pointer   to    its    display    representation 
(image  or  audio); 

(2)  a  set  of  immediate  generalizations  (super- 
concepts); 

(3)  size  if  an  image  or  duration  if  audio; 

(4)  brightness  if  an  image  or  loudness  if  audio 
(maximum  brightness  in  our  implementation); 

(5)  color  if  an  image  (each  has  a  single  color 
in  our  implementation); 

(6)  location  on  the  screen  if  an  image  or  start- 
ing time  if  audio; 

(7)  orientation  on  the  screen  if  an  image; 

(8)  periodicity  information  if  an  intermittent 
image  or  sound; 

(9)  contextual  conditions  under  which  it  is  to 
be  displayed  (see  below); 

(10)  overlay  plane  if  an  image  (see  below); 

(1 1)  a  set  of  pointers  to  its  subparts. 

And  pointers  to  these  objects  are  stored  in  a  table 
whose  retrieval  keys  are  single  facts  or  sets  of  facts. 

METUTOR  provides  two  ways  to  specify  interac- 
tions between  media  objects.  First,  as  a  simple 
method  for  handling  image  overlap,  the  teacher  can 
specify  occlusion  order  for  image  pairs.  For 
instance,  the  image  of  the  medical  corpman  could 
always  be  displayed  occluding  ("in  front  of)  the 
image  of  their  patient,  as  in  Figure  1.  Typically, 
movable    objects   should   occlude    setting-depicting 


images,  and  people  should  occlude  movable  objects. 
On  the  other  hand,  flames  and  smoke  are  more  tran- 
sparent, and  should  not  occlude  one  another  even  if 
their  images  overlap,  so  no  occlusion-order 
specification  is  needed  for  them.  Images  can  have 
holes  in  which  to  place  other  images,  like  an  outline 
of  a  tank  with  room  to  draw  a  water  level  within  it. 
(Simultaneous  audio  objects  just  have  their  signals 
added,  since  sounds  do  not  occlude.) 

The  second  kind  of  interaction  between  media 
objects  that  we  provide  for  is  contextual  variation  in 
object  graphical  properties.  This  is  common.  For 
instance,  only  when  the  firefighters  are  at  the  fire 
can  they  see  flames;  and  then  they  are  shown  at  the 
center  of  the  screen,  otherwise  to  the  right  side.  As 
another  example,  if  a  fire  team  member  gets  injured, 
that  should  reduce  by  one  the  number  of  fire  team 
members  shown  normally.  This  context  dependency 
differs  from  the  oxygen-display  example:  The  con- 
text facts  will  be  displayed  normally,  but  their  pres- 
ence causes  other  facts  to  be  displayed  differently. 
In  general,  the  presence  or  absence  of  any  fact 
could  affect  the  display  of  another. 

Surprisingly,  we  discovered  when  we  began  imple- 
menting the  preceding  ideas  that  some  natural- 
language  output  was  essential.  To  some  extent,  we 
can  avoid  it  with  metaphorical  images  and  sounds, 
like  radiating  fines  over  a  person's  head  to  indicate 
that  they  are  angry,  or  sounds  of  people  yelling. 
But  this  can  hurt  clarity:  Lines  over  a  person's  head 
could  mean  saintliness  or  a  bad  haircut.  So  we 
always  display  a  text  box  that  completely  describes 
the  current  state.  This  is  essential  for  currently- 
undisplayed  facts  (like  what  is  happening  at  the  fire 
when  you  are  not  there),  orders  that  are  currently  in 
effect,  descriptions  of  mental  states,  history,  and  so 
on,  as  well  as  clarifying  displayed  information.  To 
facilitate  natural-language  output,  we  provide  a 
modifiable  rule -based  system  for  paraphrasing  inter- 
nal representations  of  state  descriptors  and  actions. 
We  also  let  text  to  be  drawn  directly  on  the  picture, 
which  allows  labeling.  A  fact  can  be  represented  by 
multiple  pictures  and/or  multiple  texts,  like  a  picture 
of  a  switch  with  labels  on  its  positions. 

Our  current  implementation  is  in  Prolog  with 
Prowindows  and  uses  bitmaps  to  hold  all  stored 
images.  The  hardware  is  Sun  Sparc  workstations, 
and  this  does  impose  performance  limitations 
because  these  machines  are  not  designed  for  real- 
time graphics.  So  we  use  bitmaps  for  all  images 
rather  than  drawing  even  simple  shapes  in  real  time. 


Even  bitmaps  can  lake  a  second  to  load.  So  a  tech- 
nique for  additional  efficiency  that  we  have  explored 
is  to  recompute  certain  combinations  of  bitmaps  that 
are  commonly  shown  together,  "compiling"  them 
into  one  big  bitmap  which  can  be  loaded  more 
quickly  than  the  individuals. 

The  METUTOR  framework  extends  easily  to  multi- 
student  virtual  realities,  as  in  replenishment  at  sea,  a 
skill  which  requires  the  cooperation  of  two  ship 
commanders  [6].  In  our  implementation  of  this 
tutor,  each  student  runs  a  separate  program  copy 
with  its  own  goals,  and  each  program  writes  into 
and  reads  from  a  shared  state  file.  Each  program 
repeatedly  checks  the  state  file  before  changing  it; 
whenever  the  file  is  changed,  the  other  program 
throws  away  its  previous  reasoning  and  begins 
reanalyzing  the  situation  with  the  new  state. 

3.  Examples 

Figure  1  shows  an  example  from  the  tutor  previ- 
ously mentioned  for  fire  team  leaders  on  Navy 
ships,  based  on  the  analysis  of  [7].  The  text  box  is 
on  top,  the  graphical  representation  of  the  current 
state  is  below,  and  the  menu  of  actions  to  select  is 
in  the  lower  right.  Figure  1  shows  a  tutoring  ses- 
sion during  which  the  student  ordered  watching  of 
the  fire  area  for  reignition  of  the  now-extinguished 
fire,  but  he  ordered  it  before  emptying  the  compart- 
ment of  smoke,  and  has  thereby  caused  the  reflash 
watchman  to  collapse  from  smoke  inhalation.  So 
the  student  then  directed  the  medical  corpman  to 
give  first  aid  to  the  casualty.  In  the  default  color 
implementation,  the  lower-left  shapes  are  light  blue, 
the  water  is  green,  the  smoke  is  yellow,  the  medic 
on  the  right  has  a  red  cross,  the  other  people  are 
dark  blue,  and  the  other  shapes  are  black.  Note  the 
oxygen  canister  occludes  the  oxygen  tester  and  the 
medical  corpman  occludes  the  patient.  The  very 
bottom  of  the  text  box  (the  description  of  the 
current  state,  almost  identical  to  that  of  the  previous 
state  described)  has  been  obscured  to  permit  seeing 
some  of  the  previous  tutoring.  Note  that  the  tutor 
lets  students  do  foolish  things  like  this  sending  une- 
quipped people  into  smoke-filled  compartments 
because  the  visual  consequences  are  clearer  than 
words,  although  the  tutor  does  hint.  Note  also  that 
the  tutor  noticed  that  the  student  has  returned  from  a 
digression  on  their  previous  action. 

Figure  2  shows  an  example  from  the  tutor  for  emer- 
gency fuel  problems  on  F-4  aircraft  [1].  It  shows  a 
session  in  which  the  student  (a  pilot)  is  doing  some- 


thing useful,  but  not  the  most  important  action  at 
the  moment,  so  the  tutor  confines  itself  to  hints. 
Since  the  name  of  the  best  action  is  similar  to  that 
the  student  chose,  a  lexical  confusion  may  have 
occurred.  Just  four  bitmaps,  besides  the  text,  were 
used  to  create  the  entire  picture;  this  was  deliberate, 
because  real  cockpits  have  easily-confusable 
switches.  Simple  subroutines  permitted  separating 
the  shape  and  position  specifications. 

To  illustrate  declarative  teacher  specifications,  here 
are  examples  from  the  firefighting  tutor  as  they  actu- 
ally appear. 

start_state([location(repair,locker), 

raging(fire),sinokey]). 
goal([verified(out(fire)),safe(gases), 

safe(  oxygen  ),not(  equipped*  team)), 

not(smokey),not(watery), 

not(watehed(rerlashing)), 

not(unreplaced(casualty)), 

note  treated!  casualty)), 

not(dead(casualty)), 

debriefed(team), 

deenergi/ed(fire,area)|). 

The  first  two  lines  above  specify  that  in  the  starting 
state  the  fire  team  is  in  their  repair  locker,  die  fire  is 
raging,  and  the  fire  area  is  smokey.  The  remaining 
lines  specify  that  the  student  must  achieve  the  fol- 
lowing: the  fire  is  verified  to  be  out,  the  gases  and 
oxygen  in  the  fire  area  are  safe,  the  fire  team  in 
unequipped,  the  fire  area  is  neither  smokey  nor  full 
of  water,  no  reflash  watch  is  on,  no  unreplaced  or 
untreated  or  dead  casualties  are  present,  and  the  fire 
area  is  deenergized. 

recommended*  (outi  fire)  |,  extinguish), 
recommended!  |  not!  present(casualtv))], 

[  present!  medical. corpman)  |, 

directmedicalcorpman). 

The  first  says  that  if  you  want  the  fire  to  be  out,  you 
should  extinguish  it.  The  second  says  that  if  you 
want  a  casualty  to  no  longer  be  present,  then  if 
there  is  a  medical  corpman  present,  you  should 
direct  the  corpman  to  handle  the  casualty. 

precondition!  extinguish, 
|  location!  fire),  raging!  tire), 
equipped!  team  ),set(  boundaries), 
confronted(fire)]). 

This  says  that  to  extinguish  a  fire,  you  must  be  at 
the  fire,  the  fire  must  be  raging,  the  fire  team  must 
be  equipped,  the  boundaries  of  the  fire  must  be  set, 
and  you  must  be  facing  the  fire. 


deIetepostcondition(go(fire), 
[location(repair,locker)]). 

addpostconditioni  extinguish, 
[out(fire),watery,smokey]). 

addpostconditioD(extinguisb, 
[not(deenergized(fire,area))], 
[present(casualty),dead(casualty), 
present(crater),raging(fire)J, 
'There  is  a  big  explosion!'). 

The  first  definition  says  that  if  you  go  to  the  fire, 
you  are  no  longer  at  the  repair  locker.  The  second 
says  that  when  a  fire  is  extinguished,  the  fire  is  out, 
the  area  is  watery,  and  the  area  is  smokey.  The  last 
definition  says  that  in  the  special  case  where  the  stu- 
dent tries  to  extinguish  when  power  to  the  fire  area 
is  on,  a  casualty  is  present  and  dead,  a  crater  in  the 
floor  is  present,  the  fire  is  still  raging,  and  the  mes- 
sage "There  is  a  big  explosion!"  is  printed.  The  last 
definition  overrides  the  second  when  both  apply. 

randchange(extinguish,[  ], 
out(fire),raging(fire),0.3, 
'Fire  is  still  raging.'). 

This  says  that  30%  of  the  time  when  a  student  tries 
to  extinguish  a  fire,  they  fail. 

bmap(raging(fire),[location(fire)], 
fire2,308,135,red). 

This  says  that  if  the  fire  is  raging  and  the  fire  team 
is  at  the  fire,  the  screen  should  show  red  flames 
(whose  bitmap  is  in  file  "fire2")  with  upper  left 
comer  of  the  bitmap  at  (308,135).  (The  position 
was  automatically  computed  by  the  shape- 
construction  program  when  the  flames  were  created 
and  situated  in  a  dummy  box.) 

opclick(380,6,320,175, 

(location(fire),smokey],desmoke). 

This  says  that  anytime  the  student  clicks  the  left 
mouse  button  in  the  region  320  pixels  wide  and  175 
pixel  high  whose  upper  left  comer  is  at  (380,6), 
while  at  the  same  time  the  simulation  has  the  fire 
team  at  the  fire  and  the  fire  area  is  smokey,  that 
click  means  the  student  wants  to  desmoke. 

draw_order(present_casualty, 
present_medical_corpman). 

draw_order(treated_casualty, 
present_medical_corpman ). 

These  say  that  the  image  of  the  medical  corpman 
should  occlude  images  of  the  casualty  and  the 
stretcher  (the  latter  the  "treatment"). 
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